Skip to content

fix(minibf): fix ordering and 404 edge cases in account addresses - #1221

Draft
slowbackspace wants to merge 6 commits into
mainfrom
fix/minibf-account-addresses
Draft

fix(minibf): fix ordering and 404 edge cases in account addresses#1221
slowbackspace wants to merge 6 commits into
mainfrom
fix/minibf-account-addresses

Conversation

@slowbackspace

@slowbackspace slowbackspace commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Closes #1140.

Summary

Fixes two blockfrost-tests failures in GET /accounts/{stake_address}/addresses: wrong desc ordering and a 404 for pool-only accounts. Also adds a stake address log, hosted in the archive store, that serves the endpoint in O(page) time.

Rebased on main after #1301, #1302, #1303 and #1304 retired the standalone index store and redb backends. The log now lives where the other block projections live: the archive store.

Bug 1: order=desc returned the wrong order

  • Cause: the handler scanned blocks newest-first for desc and kept the first hit per address. This ordered reused addresses by their latest appearance; Blockfrost orders by first appearance (lowest tx_out.id per address, see ryo's accounts_stake_address_addresses.sql).
  • Fix: both orders are served from the stake address log below, which stores every address at its first appearance. desc is a reverse range read of the same entries, so it is the exact reverse of asc by construction.
  • Proof: the mainnet fixture account stake1u9uz4j... reuses its oldest address. All four mainnet fixture cases match a fully synced mainnet snapshot (the desc case failed before). On the synthetic chain, desc equals the reverse of asc and a desc page is a window into the reversed list.

Bug 2: pool-only accounts returned 404

  • Cause: the preprod fixture credential stake_test1uzkdwx64... appears on chain only as reward account and owner of three pools. Dolos creates no AccountState for it, so the existence guard returned 404; Blockfrost returns 200 [] because db-sync registers pool reward accounts and owners.
  • Fix: before returning 404, scan PoolState for a pool that names the credential as reward account or owner. The scan runs only on the would-be-404 path, so the hot path pays nothing.

from / to are ignored

Blockfrost declares only count, page and order for this endpoint, and neither ryo nor mimicry binds from/to in its query. MiniBF honored them through the shared pagination struct and narrowed the scan. The handler now ignores them and always covers the whole history, like the sibling handlers that take no window. accounts_by_stake_addresses_ignores_from_and_to pins this on both the log path and the scan fallback.

Stake address log

No scan can serve this endpoint. A correct scan must find every address of the account before it can order them, and it cannot tell an account with four addresses and 821k blocks from one with a fifth address in its last block. Measured on a copy of a preview store (bootstrapped 2026-09-10, five heaviest accounts by stake-tagged block count, count=100, two runs each, seconds):

Account Tagged blocks Distinct addresses asc scan desc scan (asc + reverse) log, either order (this PR)
stake_test1uz2a3lk22… 821k 4 226 226 0.0004–0.002
stake_test1ur96gg3s3… 185k 200+ 0.07 20–21 0.0004–0.002
stake_test1upv7n2x0l… 160k 200+ 0.02 15.2–15.4 0.0004–0.002
stake_test1urk948umh… 147k 3 15.2–15.6 14.9–15.7 0.0004–0.002
stake_test1uzd7tg4e6… 56k 1 8.2 7.3–7.9 0.0004–0.002

The log column comes from a preview store bootstrapped from scratch with this branch's binary (dolos bootstrap mithril, 2026-09-11). Every one of the 25 requests answered in 0.4–2.0 ms, and every response body is identical to the scan's on the older store.

A smarter scan does not change the picture. A newest-first walk that places each address at its earliest archive address tag brings the two many-address accounts to under 0.12 s for desc, but the three few-address accounts stay where they are, because only reading every block proves there is no further address. A fully synced mainnet snapshot showed 305 seconds for an exchange account with 400k+ addresses under the old scan, which is also a DoS shape on public nodes. So there is no scan fallback: the endpoint reads the log, and only the log.

The log stores each (stake credential, address) pair once, at its first on-chain appearance, ordered by (slot, tx order, output order). Both asc and desc are a single page read (desc is a reverse range read).

Piece Where Shape
Delta ArchiveIndexDelta::stake_addresses one StakeAddressAppearance { order, stake, address } per produced output with a stake credential
Write ArchiveWriter::apply_index / undo_index same call and same batch as the archive tags and exact lookups
Read ArchiveStore::addresses_by_stake_log(stake, offset, limit, reverse) one page, in either order
  • fjall: new archive-stake-log keyspace with two entry shapes: a pair entry per (stake, address) (write-path probe and undo key) and an ordered entry (page read). The writer keeps a batch-local seen-set because a batch cannot read its own pending inserts. Not swept by prune_history: entries are first appearances, so removing one below the cutoff would drop an address the account still uses.
  • memory: implements the same semantics, so ToyDomain endpoint tests exercise the log. noop answers an empty page.
  • Undo: compute_undo rebuilds the archive deltas with the same index_block the apply path uses, so undo_index removes exactly what apply_index inserted. A pair is removed only when the undone block is its stored first appearance.
  • Population: every path that applies blocks from genesis writes the log in the same batch as the blocks: relay sync, Mithril bootstrap, WAL catch-up, doctor rebuild-state, snapshot backfill. Stores that predate the log resync; that is the migration, as for every earlier index dimension.
  • Known gap, stelae: a stele restore does not write the log, because the indexes stele layer carries tag and exact records only (discriminants 0 and 1). A node bootstrapped from a stele answers this endpoint from an incomplete log until it resyncs. Closing it means a third record kind in the layer, a PROFILE_VERSION bump so old stelae are refused, and republished stelae; that is a follow-up on the stele profile, which is itself still landing (its bootstrap docs are docs: a bootstrap page for dolos bootstrap stelae #1217).
  • Size: full address bytes are stored so pointer addresses round-trip. Estimated mainnet size: 5–15GB against a 340GB store.

Testing

  • accounts_by_stake_addresses_order_desc asserts desc equals the reverse of asc and that a desc page is a window into the reversed list.
  • accounts_by_stake_addresses_pool_only_account_returns_empty_list queries the synthetic pool owner credential with no AccountState; the old implementation fails with 404.
  • accounts_by_stake_addresses_ignores_from_and_to pins that a window starting at the newest first appearance changes nothing.
  • stake_log_round_trips_and_pages in tests/archive_index_roundtrip.rs runs against fjall and memory: first-appearance dedup inside one writer and across writers, ordered paging from both ends, offset windows, in-block ordering, stake isolation, and undo of the first appearance only.
  • Full workspace suite passes; clippy clean; pinned nightly fmt applied.
  • End-to-end on a fresh preview Mithril bootstrap with this branch's binary: the log is populated through the import path, the five heaviest preview accounts answer in 0.4–2.0 ms in both orders, and all 25 bodies match the archive scan on an independently built store (see the table above).

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@michalrus

Copy link
Copy Markdown
Contributor

cargo-deny fixed in:

The /accounts/{stake_address}/addresses endpoint diverged from
Blockfrost in two cases:

- order=desc sorted addresses by their latest on-chain appearance.
  Blockfrost returns the exact reverse of the asc list, which orders
  addresses by first appearance. Reused addresses came out in the
  wrong position.

- Accounts that only appear inside pool registrations (reward account
  or pool owner) returned 404. Blockfrost knows these credentials and
  returns an empty list.

Fixes #1140
The /accounts/{stake_address}/addresses endpoint scans every archive
block that touches the account. With correct first-appearance ordering,
a desc request must scan the account's full history: 305 seconds
measured on mainnet for an exchange account with 400k+ addresses.

The stake address log stores each (stake credential, address) pair
once, at its first on-chain appearance, ordered by slot, transaction
order, and output order. Both orders become one page read.

The log is a projection of the block history, so it lives in the
archive store beside the archive tags and the exact lookups, and it is
written through the same ArchiveWriter::apply_index call in the same
batch as the blocks.

- core: `ArchiveIndexDelta::stake_addresses` carries the candidates;
  `ArchiveStore::addresses_by_stake_log` reads a page and
  `mark_stake_log_ready` declares the log complete.
- fjall: new `archive-stake-log` keyspace with pair entries (the write
  probe and undo key), ordered entries (the page read) and the ready
  marker. The writer keeps a batch-local seen-set because a batch cannot
  read its own pending inserts. The keyspace is not swept by
  prune_history: its entries are first appearances.
- memory: same semantics, so ToyDomain tests exercise the log.
- noop answers None.
- The apply path emits appearances from `index_block`. The undo path
  rebuilds the same deltas, so a rollback removes exactly what apply
  inserted, and only when the undone block was the pair's first
  appearance.
- Genesis bootstrap marks the log ready, before the state cursor, so a
  crash in between re-runs genesis. Stores restored from a stele or
  synced before the log answer None and the endpoint falls back to the
  archive scan until a resync.
@slowbackspace
slowbackspace force-pushed the fix/minibf-account-addresses branch from 568d3ba to 977e0cd Compare September 11, 2026 09:03
Blockfrost declares only count, page and order for
/accounts/{stake_address}/addresses. Neither ryo nor mimicry binds
from/to for it, so a windowed request answers the full list there.
MiniBF honored them through the shared pagination struct and narrowed
the scan, a quiet divergence.

Scan the whole history like the sibling handlers that take no window,
and drop the guard that kept windowed requests off the stake address
log. A test checks that a window starting at the newest first
appearance changes nothing on either the log path or the scan fallback.
The fallback scan for stores without the stake address log walked the
account's whole history ascending and reversed it, because a descending
walk meets a reused address at its latest appearance first. Measured on
a preview store: 20s for an account with 185k tagged blocks whose asc
page answers in 70ms.

Walk newest-first instead and ask the archive address tag where each
address really belongs: its earliest tagged block is its first
production, since an address is spent only after it is produced. Each
address is looked up once, when the scan first meets it, and emitted
when the scan reaches that block. A page of a many-address account now
fills from recent blocks and stops; the two preview accounts above drop
from 15-21s to under 0.12s with byte-identical bodies. Accounts with a
handful of reused addresses still read their whole history in either
order, as asc does on main; the stake address log is what answers those
in one page read.
The endpoint kept an archive-scan fallback for stores without the
stake address log, gated by a ready marker that genesis wrote. The
fallback is what the log exists to replace: an ascending scan that
reverses for desc reads an account's whole history, and even the
newest-first variant does so for accounts with a handful of reused
addresses (226s for the heaviest preview account). Either the index
is there or the endpoint is not worth serving from a scan.

Drop the fallback and the marker: the handler is one log read in
either order, `addresses_by_stake_log` answers a page rather than an
`Option`, and `mark_stake_log_ready` goes with its seven
implementations and the genesis hook. The log is populated by every
path that applies blocks from genesis: relay sync, Mithril bootstrap,
WAL catch-up, `doctor rebuild-state` and the snapshot backfill. A
stele restore does not write it, because the `indexes` stele layer
carries tag and exact records only; that is a known gap to close in
the stele profile.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

minibf: fix /accounts/{stake_address}/addresses

2 participants